{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow Introduction\n",
    "\n",
    "![Kubeflow Overview](img/kubeflow-overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow on AWS\n",
    "\n",
    "\n",
    "### [Blog Post:  Securing and Scaling Kubeflow on AWS](https://aws.amazon.com/blogs/opensource/enterprise-ready-kubeflow-securing-and-scaling-ai-and-machine-learning-pipelines-with-aws/)\n",
    "\n",
    "[![Kubeflow on AWS](img/kubeflow-aws-blog-post.png)](https://aws.amazon.com/blogs/opensource/enterprise-ready-kubeflow-securing-and-scaling-ai-and-machine-learning-pipelines-with-aws/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow Fairing Introduction\n",
    "\n",
    "Kubeflow Fairing is a Python package that streamlines the process of `building`, `training`, and `deploying` machine learning (ML) models in a hybrid cloud environment. By using Kubeflow Fairing and adding a few lines of code, you can run your ML training job locally or in the cloud, directly from Python code or a Jupyter notebook. After your training job is complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint.\n",
    "\n",
    "\n",
    "# How does Kubeflow Fairing work\n",
    "\n",
    "Kubeflow Fairing \n",
    "1. Packages your Jupyter notebook, Python function, or Python file as a Docker image\n",
    "2. Deploys and runs the training job on Kubeflow or AI Platform. \n",
    "3. Deploy your trained model as a prediction endpoint on Kubeflow after your training job is complete.\n",
    "\n",
    "\n",
    "# Goals of Kubeflow Fairing project\n",
    "\n",
    "- Easily package ML training jobs: Enable ML practitioners to easily package their ML model training code, and their code’s dependencies, as a Docker image.\n",
    "- Easily train ML models in a hybrid cloud environment: Provide a high-level API for training ML models to make it easy to run training jobs in the cloud, without needing to understand the underlying infrastructure.\n",
    "- Streamline the process of deploying a trained model: Make it easy for ML practitioners to deploy trained ML models to a hybrid cloud environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install kubeflow-fairing==0.7.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Restart the kernel to pick up pip installed libraries\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"<script>Jupyter.notebook.kernel.restart()</script>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "AWS_REGION_AS_SLIST=!curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/\\(.*\\)[a-z]/\\1/'\n",
    "AWS_REGION = AWS_REGION_AS_SLIST.s\n",
    "print('Region: {}'.format(AWS_REGION))\n",
    "\n",
    "AWS_ACCOUNT_ID=boto3.client('sts').get_caller_identity().get('Account')\n",
    "print('Account ID: {}'.format(AWS_ACCOUNT_ID))\n",
    "\n",
    "S3_BUCKET='sagemaker-{}-{}'.format(AWS_REGION, AWS_ACCOUNT_ID)\n",
    "print('S3 Bucket: {}'.format(S3_BUCKET))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train in the Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from kubeflow import fairing\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "def train():\n",
    "    # Genrating random linear data \n",
    "    # There will be 50 data points ranging from 0 to 50 \n",
    "    x = np.linspace(0, 50, 50) \n",
    "    y = np.linspace(0, 50, 50) \n",
    "\n",
    "    # Adding noise to the random linear data \n",
    "    x += np.random.uniform(-4, 4, 50) \n",
    "    y += np.random.uniform(-4, 4, 50) \n",
    "\n",
    "    n = len(x) # Number of data points \n",
    "\n",
    "    X = tf.placeholder(\"float\") \n",
    "    Y = tf.placeholder(\"float\")\n",
    "    W = tf.Variable(np.random.randn(), name = \"W\") \n",
    "    b = tf.Variable(np.random.randn(), name = \"b\") \n",
    "    learning_rate = 0.01\n",
    "    training_epochs = 1000\n",
    "    \n",
    "    # Hypothesis \n",
    "    y_pred = tf.add(tf.multiply(X, W), b) \n",
    "\n",
    "    # Mean Squared Error Cost Function \n",
    "    cost = tf.reduce_sum(tf.pow(y_pred-Y, 2)) / (2 * n)\n",
    "\n",
    "    # Gradient Descent Optimizer \n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) \n",
    "\n",
    "    # Global Variables Initializer \n",
    "    init = tf.global_variables_initializer() \n",
    "\n",
    "    sess = tf.Session()\n",
    "    sess.run(init) \n",
    "      \n",
    "    # Iterating through all the epochs \n",
    "    for epoch in range(training_epochs): \n",
    "          \n",
    "        # Feeding each data point into the optimizer using Feed Dictionary \n",
    "        for (_x, _y) in zip(x, y): \n",
    "            sess.run(optimizer, feed_dict = {X : _x, Y : _y}) \n",
    "          \n",
    "        # Displaying the result after every 50 epochs \n",
    "        if (epoch + 1) % 50 == 0: \n",
    "            # Calculating the cost a every epoch \n",
    "            c = sess.run(cost, feed_dict = {X : x, Y : y}) \n",
    "            print(\"Epoch\", (epoch + 1), \": cost =\", c, \"W =\", sess.run(W), \"b =\", sess.run(b)) \n",
    "      \n",
    "    # Storing necessary values to be used outside the Session \n",
    "    training_cost = sess.run(cost, feed_dict ={X: x, Y: y}) \n",
    "    weight = sess.run(W) \n",
    "    bias = sess.run(b) \n",
    "\n",
    "    print('Weight: ', weight, 'Bias: ', bias)\n",
    "    \n",
    "train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train on the Kubeflow Cluster\n",
    "\n",
    "We will show you how to run the training job in the EKS Kubeflow cluster. We use `ECR` as our container image registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Authenticate ECR\n",
    "# This command retrieves a token that is valid for a specified registry for 12 hours, \n",
    "# and then it prints a docker login command with that authorization token. \n",
    "# Then we executate this command to login ECR\n",
    "!eval $(aws ecr get-login --no-include-email --region=$AWS_REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ignore any error message below. We first check if the ECR repository already exists before creating it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an ECR repository in the same region\n",
    "!aws ecr describe-repositories --repository-names fairing-job --region=$AWS_REGION || aws ecr create-repository --repository-name fairing-job --region=$AWS_REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Setting up AWS Elastic Container Registry (ECR) for storing output containers\n",
    "# You can use any docker container registry instead of ECR\n",
    "DOCKER_REGISTRY = '{}.dkr.ecr.{}.amazonaws.com'.format(AWS_ACCOUNT_ID, AWS_REGION)\n",
    "\n",
    "fairing.config.set_builder('append', base_image='tensorflow/tensorflow:1.14.0-py3', registry=DOCKER_REGISTRY, push=True)\n",
    "fairing.config.set_deployer('job')\n",
    "    \n",
    "if __name__ == '__main__':\n",
    "    cluster_train = fairing.config.fn(train)\n",
    "    cluster_train()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# See the Completed Job in the Kubeflow Cluster\n",
    "Re-run the cell above, if you don't see `fairing-job` below.  The fairing-job will get cleaned up after a few seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
